﻿ Comparing discourse tree structures Elena Mitocariu1, Daniel-Alexandru Anechitei1, Dan Cristea1,2 1 “Al I Cuza” University of Iasi, Faculty of Computer Science 16, General Berthelot St , 700483 – Iasi, Romania 2 Romanian Academy, Institute for Computer Science 2, T Codrescu St , 700481 – Iasi Romania {elena mitocariu, daniel anechitei, dcristea}@info uaic ro Abstract The existing discourse parsing systems make use of different theo- ries to put at the basis of processes of building discourse trees Many of them use Recall, Precision and F-measure to compare discourse tree structures These measures can be used only on topologically identical structures However, there are known cases when two different tree structures of the same text can express the same discourse interpretation, or something very similar In these cases Pre- cision, Recall and F-measures are not so conclusive In this paper, we propose three new scores for comparing discourse trees These scores take into consid- eration more and more constraints As basic elements of building the discourse structure we use those embraced by two discourse theories: Rhetorical Structure Theory (RST) and Veins Theory, both using binary trees augmented with nuclearity notation We will ignore the second notation used in RST – the name of relations The first score takes into account the coverage of inner nodes The second score complements the first score with the nuclearity of the relation The third score computes Precisions, Recall and F-measures on the vein expressions of the elementary discourse units We show that these measures reveal compa- rable scores there where the differences in structure are not doubled by differ- ences in interpretation Keywords: discourse parser, Rhetorical Structure Theory, Veins Theory, eval- uation, discourse tree structure 1 Introduction Discourse parsing systems have an important role in many NLP applications such as question-answering, summarization or deep understanding of texts Driven by one or another of the discourse theories, the majority of parsers produce tree structures However, although correctly revealing relations between parts of the discourse, the topology of trees sometimes evidence differences which are not anchored in true dis- course facts To give just one example, a narration (sequence of elementary discourse units of equal salience) could be represented by binary trees in many ways This paper proposes new measures of evaluating discourse trees which shadows idiosyncrasies of notation, while bringing forth differences which are supported by discourse phenome- na We propose three scores for comparing the discourse trees and we exemplify them from the point of view of Rhetorical Structure Theory and Veins Theory The first score takes into account only the coverage of the nodes The second score takes into consideration the nuclearity of the relations The third score computes Precisions, Recall and F-measures on vein expressions These metrics can be applied to either RST or VT annotated trees Among the theories dealing with the organization of the discourse, Rhetorical Structure Theory (RST) is well accepted and, when the discourse structure should be seen in correlation with referentiality, Veins Theory (VT) proposes to look at the discourse structure from another angle Both use binary tree1 representations, in which leafs are elementary discourse units (edus), such as clauses or short sentences, and internal nodes represent larger text spans RST uses a labeling function that attaches relation names and nuclearities to its inner nodes, while VT ignores the names of relations Because of this simplified representation, one can say that VT is included in RST For instance, on an annotated RST corpus one can calculate head the vein ex- pressions, which are of primary importance only in VT However, the conclusions that VT develops go in a different direction then those set by RST To compare discourse structures, the PARSEVAL measures , based on Preci- sion, Recall and F-measure, are used The main discontent is that these measures can- not be used to account for structures which, although different, have rather similar interpretations In the following, we will consider binary discourse trees on which parent nodes specify the nuclearities of the left and right daughters For example, in Fig 1 the node labeled 1 is a nucleus and the node labeled 2 is a satellite Fig 1 The discourse tree representation The automatic identification of discourse structure should include, as a preliminary step, a method for the segmentation of the discourse in edus Discourse tree structures are the result of discourse parsing systems and many of them are developed based on the RST terminology Since Marcu’s first attempt at developing a rule-based dis- 1 A binary tree is a tree in which every node has at most two children The binary trees used to represent discourse structures are always strict (proper) binary trees (each parent node has exactly two children) course parser on the base of discourse markers, several algorithms for discourse parsing have been proposed, both statistical and rule-based The SPADE algorithm doesn’t use cue words for identification of rhetorical relations , using instead syn- tactic and lexical fetures It is a sentence-level discourse parser which exploits a set of lexico-gramatical dominance relations corresponding to the syntactic representation of sentences In is described an algorithm that scores and labels relations between spans, where the discourse parse trees are the result of a supervised leaning A proba- bilistic parser for parsing the discourse structure of dialogue is presented in In the following, we make first a brief presentation of the two theories we have used to exemplify the structure of discourse: Rhetorical Structure Theory – in section 2, and Veins Theory – in section 3, by insisting only on notions linked to representa- tional issues In section 4 we introduce and discuss the scores and in section 5 we give some conclusions 2 Rhetorical Structure Theory Rhetorical Structure Theory developed by Thompson and Mann is one of the most used theories among those dealing with the structure of discourse It is used in dis- course analysis and text generation Like in the majority of discourse theories, RST segments the text in edus, which could be clauses or sentences expressing ele- mentary events, situations, statements In RST, the spans of the text have a different importance in the whole discourse Nuclei are the most important spans in the text and are compulsory for the coherence of the discourse When they are edus, their deletion makes the text incoherent When they are larger than edus, at least parts of them should be kept if a simplification is intended Unlike nuclei, satellites only give addi- tional information They complement the statements/situations/events communicated by the nuclei and, if deleted from the text, the overall understability will not be harmed, even if some details will be missing Thus, the discourse includes nucleus-satellite relations and multinuclearity rela- tions In a nucleus-satellite (hypotactic) relation, the whole span includes exactly one nucleus and any number of satellites In a multinuclear (paratactic) relation, all subspans have the same importance and are considered nuclear Although, the RST trees are not necessarily binary, equivalent transformations can be imagined that make them binary For instance, a 3-arguments paratactic relation can be transformed into 2 binary paratactic relations with the same name 3 Veins Theory Taking from RST the notion of nucleus and satellite, but ignoring the name of the relations, VT attaches two expressions to each node of a binary discourse tree: the head and the vein The head expression is a list of the most salient edus that the node covers and is calculated bottom-up, as follows: ─ if the node is a leaf, the head expression is its label; ─ else, the head expression is the concatenation of the head expressions of its nu- clear children By definition, the vein expression of a node n is that list of elementary discourse units which are of primary importance for understanding the span the node n covers in the context of the whole discourse It is calculated top-down, as follows: ─ the vein expression of the root node is its head expression; ─ otherwise, if the node is a nucleus and his father has the vein expression v, then:  if the node has a left satellite sibling with head h, then its vein expression is seq(mark(h), v);  else: v; ─ finally, if the node is a satellite with the head h and the vein of his father is v, then:  if it is a left son, its vein expression is seq (h,v);  else its vein expression is (seq(h,simpl(v)) The seq, simpl, mark, are function which can be described thus: ─ seq returns the concatenation of two sequences of nodes, considering the fron- tier ordering, from left to right; ─ mark returns the same symbols as in its argument, but marked in some way (for example between parentheses or primed); ─ simpl eliminates all marked symbols from its argument 4 The Scores In discourse representation theories, it is notorious that a certain vision on a discourse tree structure can be mirrored by different tree topologies, which not necessarily ex- press significant differences in the interpretation As such, because representing dis- course structures by trees has become a standard in many theories (SDRT, RST, VT, etc ), it would be useful to have a measure by which to compare these structures and show to what extend they mean the same thing or not In the technology of discourse parsing, this issue is particularly important for at least three reasons First, a corpus of gold files should be prepared by hand, using the competencies of human annotators And, because the appreciation of a discourse structure is to such a large extend sub- jective, we would like to know the opinions of more experts on the issue Then, whenever corpora are built in NLP and more annotators are involved, their outcomes should be compared for agreement Second, evaluation of a discourse parser is usually done by comparing the parser’s output against this gold representation contributed by humans Finally, modifying certain parameters of the parser could yield trees which are slightly different, while most of the rest of the structure could be shared But how much identical are these trees? Where in the structure of the tree, differences start to be significant? To what extend some differences can be ignored, therefore character- izing stable decisions of the parser? In this section we will propose a number of scores that are intended to offer better comparisons between discourse tree structures There are many possibilities to represent by binary trees a discourse structure The number of different binary trees that can be drawn with n terminal nodes is given by the Catalan numbers : (1) For example, in case of a discourse with 4 edus, formula (1) above indicates 5 dif- ferent binary trees, which are depicted in (Fig 2) 4 1 The Overlapping Score In it is shown that a discourse tree can be formally represented as a set of tuples of the form Rk[i,m,j], denoting a relation and its two arguments, as text spans: the left one spanning between edus i and m, and the right one spanning between edus m+1 and j For example, in the tree of Fig 2(c) the tuples are : R1 , R2 and R3 The tuple representing the root relation will cover all leafs The Overlapping Score (OS) is taking into consideration only the coverage of the nodes The representation with tuples can be used for that By enduring an abuse of language, we will say that two tuples overlap if their starting and ending positions are identical (i1=i2 and j1=j2) With this convention, the score will be computed by divid- ing the number of overlapping tuples of a discourse tree structure of the same text, to the total numbers of tuples (number of leafs -1) as presented in equation (2): (2) At least one pair of tuples are overlapping, the ones describing the root relation and which should cover the whole range of nodes Two different tree structures with the same text segmentation in clauses will have the same number of leafs To exemplify OS, we will compute it for some pairs of trees in the family of trees of (Fig 2) (imagining that one represents a gold file and the second is contributed by a discourse parser) For example if we consider the discourse trees from Fig 2(a) and Fig 2(b), the number of overlapping tuples is two: R3 overlaps with R2 (covering nodes: 1 and 2) and R1 is identical with R1 (covering nodes 1, 2, 3, 4) The total number of tuples for this example is 3 The remaining pair of tuples (R2 and R3 are not overlapping because i1 i2 and also j1 j2 Applying equation (2) we obtain OS = 2/3 If we analyze the trees from Fig 2(a) and Fig 2(c), the number of overlapping tu- ples is 1 and the OS will be 1/3 Thus we can conclude that the discourse tree struc- ture from Fig 2(a) is more similar to the discourse tree structure of Fig 2(b) than to that in Fig 2(c) If OS equals 1 it means that the two discourse trees structures are identical from the point of view of the topology Fig 2 Different representations as binary trees for a text with 4 edus 4 2 The Nuclearity Score As known, in both RST and VT, the nuclearity is very important Based on OS, we can develop a Nuclearity Score (NS) This will take into consideration, besides the overlapping spans under nodes, as given by the OS, the nuclearity of the relations for the corresponding overlapping tuples NS is calculated by multiplying OS with the ratio between the number of overlap- ping tuples with identical nuclearity (let’s call these nuclearity-overlapping tuples) and the total number of overlapping tuples (3) After applying formula (2) in (3) and simplification with #overlapping tuples, for- mula (4) is obtained: (4) The difference between Fig 2 and Fig 3 is that the last one includes also marking for the nuclearity of the parrent nodes To exemplify how NS is computed, let’s con- sider the discourse trees of Fig 3(b) and Fig 3(c): there are two pairs of overlapping tuples: (N S – N S ) and (N S – S N ), out of which only one pair is also a nuclearity-overlapping tuple: (N S – N S ) Thus, for- mula (4) will give NS = 1/3 Fig 3 Examples of tree representations including nuclearity markings Like in the previous case, if the score is 1, it means that both trees are identically in terms of the topology but also the nuclearity The way the two scores have been de- fined makes that the Nuclearity Score be less then or equal with the Overlapping Score for the same pair of trees, showing that NS is more restrictive than OS When NS is significantly lower than OS, this will highlight differences in nuclearity Let’s note also that nuclearity is relevant only for sub-structures that overlap An indicator like NS is important in applications like summarization, where nuclei are relevant As an observation, considering the fact that the scores depend on each other, this means that if OS is nil then NS will be also nil 4 3 The Vein Scores The third proposal we make for comparing discourse structures takes into considera- tion Precisions, Recall and F-measures on vein expressions Vein expressions can be computed only on discourse trees that display the nuclearity of nodes As shown in Sections 2 and 3 above this can be done on both RST and VT annotated trees and the computation proceeds as was presented in Section 3 above VT uses the vein expres- sions to measure the coherence of discourse segments as well as of the global dis- course As such, Vein Scores, in addition to the other two scores defined earlier, place centrally the possibility to compare discourses on the ground of coherence, in the measure in which this is implicitly incorporated in tree structure The formulas are as follows: - (5) - (6) - - - (7) - - ILi - represents the number of identical labels from the vein expression of the edu i in the test tree and the gold tree; Ti - represents the total number of labels of the vein expression for edu i in the test tree; Gi - represents the total number of labels of the vein expression for edu i in the gold tree; N - represents the total number of edus Fig 4 Two discourse tree representations including the head and vein expressions for a text with three elementary discourse units Let us consider the two trees in Fig 4, with three edus in which the head (H) and vein (V) expressions are noted Suppose that the discourse tree of Fig 4(a) is a test tree and the tree of Fig 4(b) is the gold tree Applying formulas (5), (6) and (7) we obtain: Although the trees from Fig 4 are different in terms of structure and nuclearity, it can be seen that the vein expressions of the edus of both trees are identical, which means that they convey the same meaning Let us consider another example: Fig 5 Two discourse tree representations including the head and vein expressions for a text with four elementary discourse units Applying the same formulas (5), (6) and (7) we obtain the following results: From Fig 5 it can be seen that the two trees are totally different in terms of topolo- gy and nuclearity, but two of the 4 edus display the same vein expressions Corre- spondingly, the scores computed above reflect this resemblance If the first two scores are relevant for comparing discourse tree structures, this one is more relevant for summarization applications Thus we develop a method of com- paring discourse trees based on three scores beginning from the discourse structure to the summarization application, considering both Rhetorical Structure Theory and Veins Theory 5 Conclusion We have proposed three new scores for comparing discourse tree structures The first one measures the identical coverings among all substructures of the discourse struc- ture, the second one factors this number with identical nuclearities and the third one compares vein expressions of the terminal nodes The idea behind these scores is to consider less distant tree structures that could manifest differences in topology if these differences are not relevant for the interpretation of the discourse Also, our measures make possible to evaluate test structures produced by a parser even against gold struc- tures annotated with a different theory than the one the parser relies on Moreover, different parsers can now be evaluated and compared better, because not relevant idiosyncrasies of discourse structures pass now unnoticed by the scores The three measures connect well with different interests that could motivate the comparison of structures One of the applications in which they are most useful is summarization, where the nuclearity of the units has a great importance in recogniz- ing the main ideas References 1 Roark, B , Harper, M , Charniak, E , Dorr, B , Johnson, M , Kahne, J G , Liuf, Y , Ostendorf, M , Hale, J , Krasnyanskaya, A , Lease, M , Shafran, I , Snover, M , Stewart, R , Yung, L : SParseval: Evaluation metrics for parsing speech In: Proceedings of LREC (2006) 2 Marcu , D : The theory and Practice of Discourse Parsing and Summarization MIT press (2000) 3 Soricut, R , Marcu, D : Sentence Level Discourse Parsing using Syntactic and Lexical In- formation In: Proceedings of the Human Language Technology and North American As- sociation for Computational Linguistics Conference, pp 149-156, Edmonton (2003) 4 Hernault, H , Prendinger, H , duVerle, D , Ishizuka, M : HILDA: A Discourse Parser Using Support Vector Machine Classification Dialogue and Discourse, pp 1-33 (2010) 5 Reitter, D : Simple signals for complex rhetorics : On rhetorical analysis with rich-features support vector models LDV-Forum, GLDV-Journal for Computational Linguistics and Language Technology, pp 38-52 (2003) 6 Baldridge, J , Lascarides, A : Probabilistic head-driven parsing for discourse structure In : Proceedings of the Ninth Conference on Computational Natural Language Learning, pp 96- 103 Ann Arbor, Michigan (2005) 7 Mann, W C , Thompson, S A : Rhetorical Structure Theory: Toward a functional theory of text organization Text, pp 243-281 (1988) 8 Taboada, M , Mann, W C : Rhetorical Structure Theory: looking back and moving ahead Discourse Studies, pp 423 – 459 (2006) 9 Cristea, D , Ide, N , Romary, L : Veins theory: A model of global discourse cohesion and coherence In: Proceedings of the 17th international conference on Computational linguis- tics, pp 281-285, Montreal (1998) 10 Davis, T : Catalan Numbers, http://www geometer org/mathcircles 